%\subsubsection{Register nodes for provisioning}

\noindent Next, we add {\em compute} nodes and define their properties as
objects in \xCAT{} database. 
These hosts are grouped logically into a group named {\em
compute} to facilitate group-level commands used later in the recipe.  Note the
use of variable names for the desired compute hostnames, node IPs, MAC
addresses, and BMC login credentials, which should be modified to accommodate
local settings and hardware. To enable serial console access via  \xCAT{}, 
{\texttt serialport} and {\texttt serialspeed}
properties are also defined.

% begin_ohpc_run
% ohpc_validation_newline
% ohpc_validation_comment Add hosts to cluster \ref{sec:xcat_add_nodes}
\begin{lstlisting}[language=bash,keywords={},upquote=true,basicstyle=\footnotesize\ttfamily,]
# Define nodes as objects in xCAT database
[sms](*\#*) for ((i=0; i<$num_computes; i++)) ; do
		mkdef -t node ${c_name[$i]} groups=compute,all ip=${c_ip[$i]} mac=${c_mac[$i]} netboot=xnba \
		arch=x86_64 bmc=${c_bmc[$i]} bmcusername=${bmc_username} bmcpassword=${bmc_password} \
		mgt=ipmi serialport=0 serialspeed=115200 
        done
\end{lstlisting}
% end_ohpc_run

\begin{center}
  \begin{tcolorbox}[]
    \small 
Defining nodes one-by-one, as done above, is only efficient
for a small number of nodes. For larger node counts,
\xCAT{} provides capabilities for automated detection and
configuration.
Consult the
\href{http://xcat-docs.readthedocs.io/en/stable/guides/admin-guides/manage_clusters/ppc64le/discovery/}{\color{blue}\xCAT{}
Hardware Discovery \& Define Node Guide}. Alternatively, confluent, a tool
related to \xCAT{}, also has robust discovery capabilities and can be used to
\href{https://hpc.lenovo.com/users/documentation/confluentdisco.html}{\color{blue}detect
and auto-configure} compute hosts.  
\end{tcolorbox}
\end{center}

\iftoggleverb{isxCATstateful}
Next, we set root password for {\em compute} nodes. Note that \xCAT{} sets up
\texttt{ssh} keys to enable password-less logins for root across the cluster,
but the password is required for kickstarting the stateful install.

% begin_ohpc_run
% ohpc_validation_comment Set default root password
\begin{lstlisting}[language=bash,keywords={},upquote=true,basicstyle=\footnotesize\ttfamily,]
[sms](*\#*) chtab key=system passwd.username=root passwd.password=`openssl rand -base64 12`
\end{lstlisting}
% end_ohpc_run
\fi

\noindent \xCAT{} requires a network domain name specification for system-wide name
resolution. This value can be set to match your local DNS schema or given a
unique identifier such as ``local''. In this recipe, we leverage the
\texttt{\$domain\_name} variable to define as follows:

% begin_ohpc_run
% ohpc_validation_newline
% ohpc_validation_comment Define local domainname
\begin{lstlisting}[language=bash,keywords={},upquote=true,basicstyle=\footnotesize\ttfamily,literate={BOSVER}{\baseos{}}1]
[sms](*\#*) chdef -t site domain=${domain_name}
\end{lstlisting}

%\clearpage
If enabling {\em optional} IPoIB functionality (e.g. to support Lustre over \InfiniBand{}), additional
settings are required to define the IPoIB network with \xCAT{} and specify
desired IP settings for each compute. This can be accomplished as follows for
the {\em ib0} interface:

% begin_ohpc_run
% ohpc_validation_newline
% ohpc_validation_comment Setup IPoIB networking
% ohpc_command if [[ ${enable_ipoib} -eq 1 ]];then
% ohpc_indent 5
\begin{lstlisting}[language=bash,keywords={},upquote=true,basicstyle=\footnotesize\ttfamily]
# Define ib0 netmask
[sms](*\#*) chdef -t network -o ib0 mask=$ipoib_netmask net=${c_ipoib[0]}

# Enable secondary NIC configuration
[sms](*\#*) chdef compute -p postbootscripts=confignics

# Register desired IPoIB IPs per compute
[sms](*\#*) for ((i=0; i<$num_computes; i++)) ; do
		chdef ${c_name[i]} nicips.ib0=${c_ipoib[i]} nictypes.ib0="InfiniBand" nicnetworks.ib0=ib0
        done
\end{lstlisting}
% ohpc_indent 0
% ohpc_command fi
% end_ohpc_run

%\clearpage
With the desired compute nodes and domain identified, the remaining steps in the
provisioning configuration process are to define the provisioning mode and
image for the {\em compute} group and use \xCAT{} commands to complete
configuration for network services like DNS and DHCP. These tasks are
accomplished as follows:

%\clearpage
% begin_ohpc_run
% ohpc_validation_newline
% ohpc_validation_comment Complete networking setup, associate provisioning image
\begin{lstlisting}[language=bash,keywords={},upquote=true,basicstyle=\footnotesize\ttfamily,literate={BOSVER}{\baseos{}}1 {IMAGE}{\installimage{}}1]

# Complete network service configurations
[sms](*\#*) makehosts
[sms](*\#*) makenetworks
[sms](*\#*) makedhcp -n
[sms](*\#*) makedns -n

# Associate desired provisioning image for computes
[sms](*\#*) nodeset compute osimage=BOSVER-x86_64-IMAGE-compute
\end{lstlisting}

%%% If the Lustre client was enabled for computes in \S\ref{sec:lustre_client}, you
%%% should be able to mount the file system post-boot using the fstab entry
%%% (e.g. via ``\texttt{mount /mnt/lustre}''). Alternatively, if 
%%% you prefer to have the file system mounted automatically at boot time, a simple
%%% postscript can be created and registered with \xCAT{} for this purpose as follows.
%%% 
%%% % begin_ohpc_run
%%% % ohpc_validation_newline
%%% % ohpc_validation_comment Optionally create xCAT postscript to mount Lustre client
%%% % ohpc_command if [ ${enable_lustre_client} -eq 1 ];then
%%% % ohpc_indent 5
%%% \begin{lstlisting}[language=bash,keywords={},upquote=true,basicstyle=\footnotesize\ttfamily,literate={BOSVER}{\baseos{}}1]
%%% # Optionally create postscript to mount Lustre client at boot
%%% [sms](*\#*) echo '#!/bin/bash' > /install/postscripts/lustre-client
%%% [sms](*\#*) echo 'mount /mnt/lustre' >> /install/postscripts/lustre-client
%%% [sms](*\#*) chmod 755 /install/postscripts/lustre-client
%%% # Register script for computes
%%% [sms](*\#*) chdef compute -p postscripts=lustre-client
%%% \end{lstlisting}
%%% % ohpc_indent 0
%%% % ohpc_command fi
%%% % end_ohpc_run
%%% 

